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Abstract 

Background: "Cumulative meta-analysis" describes a statistical procedure to calculate, retrospectively, summary estimates 
from the results of similar trials every time the results of a further trial in the series had become available. In the early 1990s, 
comparisons of cumulative meta-analyses of treatments for myocardial infarction with advice promulgated through medical 
textbooks showed that research had continued long after robust estimates of treatment effects had accumulated, and that 
medical textbooks had overlooked strong, existing evidence from trials. Cumulative meta-analyses have subsequently been 
used to assess what could have been known had new studies been informed by systematic reviews of relevant existing 
evidence and how waste might have been reduced. 

Methods and Findings:\Ne used a systematic approach to identify and summarise the findings of cumulative meta-analyses 
of studies of the effects of clinical interventions, published from 1992 to 2012. Searches were done of PubMed, MEDLINE, 
EMBASE, the Cochrane Methodology Register and Science Citation Index. A total of 50 eligible reports were identified, 
including more than 1,500 cumulative meta-analyses. A variety of themes are illustrated with specific examples. The studies 
showed that initially positive results became null or negative in meta-analyses as more trials were done; that early null or 
negative results were over-turned; that stable results (beneficial, harmful and neutral) would have been seen had a meta- 
analysis been done before the new trial; and that additional trials had been much too small to resolve the remaining 
uncertainties. 



Conclusions: This large, unique collection of cumulative meta-analyses highlights how a review of the existing evidence 
might have helped researchers, practitioners, patients and funders make more informed decisions and choices about new 
trials over decades of research. This would have led to earlier uptake of effective interventions in practice, less exposure of 
trial participants to less effective treatments, and reduced waste resulting from unjustified research. 
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Introduction 

In 1992, a team lead by Tom Chalmers and Fred Mosteller 
introduced the term 'cumulative meta-analysis' to describe a 
statistical procedure to calculate, retrospectively, summary esti- 
mates based on the results of similar trials every time the results of 
a further trial in the series had become available [1]. One of their 
two papers published in 1992 made clear how important this 
procedure was for auditing both research and healthcare advice. 
Comparisons of the results of cumulative meta-analyses of 
treatments for myocardial infarction with the advice that had 
been promulgated through medical textbooks [2] made clear not 
only that research had continued long after robust estimates of 
treatment effects had accumulated, but also that medical textbooks 
had overlooked strong, existing evidence from clinical trials, both 
of beneficial and of lethal effects of treatments [3] . 



Cumulative meta-analyses have subsequently been used to 
assess what could have been known had the design of new studies 
been informed by reference to systematic reviews of relevant 
existing evidence and how these might have reduced waste [4]. 
Cumulative meta-analyses emphasise the need for the design of 
new studies to be informed by existing research [5] and for the 
results of new studies to be set in the context of updated systematic 
reviews of the relevant evidence from all sufficiently similar studies 
[6]. The idea of using the accumulating evidence to make 
decisions about the design and ongoing conduct of trials is not 
new: the report of a case study published by Henderson and 
colleagues nearly 20 years ago noted "Our thesis is that if related 
published trials are available, a meta-analysis should be started in 
the planning stages of a clinical trial, continued through the 
ongoing conduct of the trial, and performed as one analysis among 
many in the final analysis of the trial" [7]. Such reviews and meta- 
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analyses help to provide the ethical, scientific and environmental 
justification for both new study and for any future studies [8] . 

In this methodological review, we use systematic methods to 
search for and summarise the findings of cumulative meta-analyses 
of studies of the effects of clinical interventions, published from 
1992 to 2012. We describe the different settings for these studies 
and explore their findings in the context of unnecessary 
duplication of effort or waste if trials were done after a robust 
finding would have been found if a review and meta-analysis of 
existing research had been performed. By conducting this research 
as a systematic review, our aim is to provide the most 
comprehensive collection of cumulative meta-analysis of studies 
of healthcare interventions. The searching for this review also 
identified several cumulative meta-analyses in other types of health 
research, which are not summarised here but have been discussed 
in brief elsewhere [4]. For example, if epidemiological studies 
investigating possible aetiological factors in sudden infant death 
syndrome had taken proper account of the accumulating evidence, 
the lethal effect of 'front lying' would have been recognized at least 
a decade earlier, and tens of thousands of infant deaths could have 
been avoided [9] . A cumulative meta-analysis of 55 studies that 
continued to be conducted over more than two decades showed 
that for over 1 7 years there had been ample evidence that never- 
smoking women who had been exposed to spousal smoking were 
more likely than controls to develop lung cancer [10]. 

Methods 

Inclusion criteria 

Studies were eligible if they included a cumulative meta-analysis 
of studies of the relative effects of alternative healthcare 
interventions. Ideally, the cumulative meta-analysis would be 
presented as a graph showing the summary estimates as each 
study's result was added to the meta-analyses in the order this had 
been published or became available in some other way. However, 
studies were eligible if these graphs or the separate summary 
estimates were not available, but the general findings or 
implications of the cumulative meta-analyses were reported. 
Studies were eligible if a single cumulative meta-analysis was 
presented, or if a meta-epidemiological project had been done in 
which numerous systematic reviews or meta-analyses were 
included. Eligible cumulative meta-analyses had sometimes been 
done as part of the planning process for a new study to explore 
how the evidence base had evolved. Studies were not included if 
only surrogate outcome measures were used unless - like blood 
pressure and severe anaemia, for example - these were unambig- 
uously important. The searches also identified cumulative meta- 
analysis in other types of health research (including observational 
epidemiology and genetics) but these were not eligible for this 
review. 

Search strategy 

We wished to identify published reports containing graphs of 
cumulative meta-analyses. An initial search of Pubmed Clinical 
Queries using the term 'cumulative meta-analysis' retrieved 822 
records. The selected papers were screened by one author (IC), 
who selected a set of 23 records which were added to a set of core 
articles that had been previously identified as part of this author's 
general interest in this area. This full set of papers were then 
analysed to obtain ideas for free-text search terms and index or 
MeSH terms for use in a final search strategy. 

Searches were then run on the following databases: MEDLINE, 
OVID (1946-2012, In-Process and other non-indexed citations); 
EMBASE, OVID (1947-2012); the Cochrane Methodology 



Register (2012); and Science Citation Index (2012). No language 
or publication date restrictions were applied. The following search 
strategy was used for MEDLINE: cumulative adjlO metapana- 
lys$.mp. OR (cumulative. mp. and meta-analysis.pt.). The search of 
the Cochrane Methodology Register combined the output of a 
simple search for the word 'cumulative' in the abstract with that 
for a search for records that had been assigned the term 'Meta- 
analysis updating and cumulative meta-analysis' by the compilers 
of the Register. The searches were conducted in June 2012. 

The reference lists of relevant systematic reviews and other 
publications were checked to identify additional articles [11]. A 
search was undertaken in Google Scholar to identify relevant 
terms in the full text of articles where relevant concepts had not 
been located in bibliographic databases using searches of tides, 
abstracts and index terms. 

Assessment of studies, data extraction and analysis 

Records were exported from Endnote to Sente for first 
screening by one of the authors (AB). This screening was 
deliberately over-inclusive, and all potentially eligible articles or 
unclear articles were checked by a second author (IC). The 
eligibility of all studies arising from this process was then 
confirmed by a third author (MC) before their inclusion in this 
review. 

The process of data extraction was piloted in ten records and 
emerging themes identified. A final data extraction form was then 
agreed. All eligible papers were read, and passages of text 
commenting on the details of the cumulative meta-analysis and the 
implications of the completed cumulative meta-analyses were 
identified and extracted. This was done by one author (AB) and 
the data extracted were checked against the original articles by the 
other authors (IC and MC). 

No attempts were made to combine the results of the included 
studies, because the aim of this review is to present a range of 
examples and identify themes across the cumulative meta-analyses 
identified, rather than to attempt to generate a statistical finding. 
Similarly, this review does not seek to estimate the incidence or 
prevalence of trials that have been conducted unnecessarily, since 
that would be done better with a comprehensive study of large 
cohorts of trials addressing the same or similar questions. Instead, 
it seeks to identify and report a range of examples to examine 
whether cumulative meta-analyses might have led to different 
choices in the conduct of new research. 

Results 

Results of the search 

This initial screening identified 942 records needing inspection, 
of which 8 1 8 were excluded in the initial screening. This screening 
identified 46 records for inclusion and 78 needing further review. 
The second screening step excluded 74 of these, leaving 50 reports, 
including more than 1500 cumulative meta-analyses of clinical 
intervention studies, as eligible. 

Included studies 

The included studies were published from 1992 to 2012 
inclusive. Most of these studies focused on a single question for a 
systematic review or meta-analysis, and included just one graphical 
presentation of a cumulative meta-analysis. The 50 studies 
addressed a wide range of health problems. Fourteen covered 
aspects of surgery for a variety of problems (reducing infection and 
bleeding and improving surgical technique). Eleven studies 
concerned heart disease, and there were at least two for each of 
stroke, neonatal problems, infectious disease, cancer, mental 
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health, dental health and respiratory illness. There were single 
studies in obstetrics, musculoskeletal disease, kidney disease and 
gastrointestinal disease. The number of individual research studies 
in each of the cumulative meta-analyses we identified ranged from 
4 trials of continuous versus interrupted techniques for elective 
midline abdominal fascial closure [12] to more than 60 trials for 
aprotinin in cardiac surgery [13]. 

Nearly all of the cumulative meta-analyses had been conducted 
as retrospective exercises to explore how the evidence base had 
evolved. However, some were used to inform the design of a new 
study. For example, Algra and van Gijn presented a cumulative 
meta-analyses which was used to inform the need for the 
European and Australian Stroke Prevention in Reversible 
Ischaemia Trial (ESPRIT) [14]. 

A variety of themes were apparent from the cumulative meta- 
analyses and these are illustrated here with specific examples. A 
detailed explanation is not provided for each cumulative meta- 
analyses, but the citations for all the included studies are available 
in Appendix S 1 . The included studies showed that initially positive 
results became null or negative in the meta-analysis as more trials 
were done; that early null or negative results were over-turned; 
that stable results (beneficial, harmful and neutral) would have 
been seen had a meta-analysis of existing evidence been done 
before the new trial; and that additional trials had been much too 
small to resolve the remaining uncertainties. 

Positive results becoming less so 

Cumulative meta-analyses have shown how replications have 
challenged initially favourable results [15-18] where the early 
trials were favourable but not statistically significant. For example, 
Klein and colleagues examined the effects of cognitive behavioural 
therapy (CBT) for adolescent depression, using the publication 
dates for 1 1 randomised trials to build a cumulative meta-analysis. 
They found that the effect size from the meta-analyses had 
decreased steadily from the large effects observed in the earliest 
trials, with narrowing confidence intervals as the data accumulat- 
ed. However, they also noted that the trend could be related to 
methodological differences between the studies and wrote "these 
differences appear to reflect both a shift from an initial emphasis 
on demonstrating the efficacy of treatment in controlled research 
settings to an emphasis on demonstrating the effectiveness of 
treatment and the application of increased statistical and 
methodological rigor over time". They concluded that "the results 
indicate that CBT may be effective for the acute treatment of 
depression among adolescents, although treatment effects may be 
more modest in clinical settings than findings from early trials 
would suggest" [17]. 

Null or negative results becoming positive 

Cumulative meta-analyses have also shown how replications 
have sometimes challenged initially unfavourable results [19—20], 
but these examples are fewer and weaker than for the previous 
theme. For example, in a systematic review comparing plating 
versus intramedullary nailing of humeral shaft fractures in adults, 
the cumulative meta-analyses by Li and colleagues showed that 
intramedullary nailing might increase the re-operation rate in 
studies conducted before 2000 (odds radio (OR): 0.39, 95% 
confidence interval (CI): 0.17 to 0.90, P = 0.03). They noted that 
although the point estimate was still unfavourable, the difference 
was not statistically significant in studies conducted after that year 
(OR: 0.54, 95% CI: 0.27 to 1.08, P = 0.08) [19]. 



Stable results 

This collection of cumulative meta-analyses includes many that 
show that a systematic review of existing research would have 
reduced uncertainty about an intervention. They showed how 
morbidity and mortality might have been reduced, both for 
patients in general and for participants in trials in which they may 
have continued to be allocated to a placebo or a control group 
when the intervention could have been shown to be effective in a 
meta-analysis [1-2, 12-14,18,21-38]. This observation also has 
implications for waste in research, since new studies might have 
been regarded as unjustified if a systematic review and meta- 
analysis had been done as part of its design phase. Two examples, 
through two decades, are used to illustrate this. 

In 1992, Lau and colleagues reported a collection of cumulative 
meta-analyses of clinical trials that evaluated 15 treatments and 
preventive measures for acute myocardial infarction. They found a 
consistent, statistically significant reduction in total mortality for 
the use of streptokinase (OR: 0.74, 95% CI: 0.59 to 0.92) would 
have been shown in 1973, after only eight trials involving 2432 
patients had been completed. The results of 25 subsequent trials, 
which enrolled an additional 34,542 patients through 1988, had 
littie or no effect on the odds ratio establishing efficacy, but 
narrowed the 95% confidence interval [1]. 

The cumulative meta-analysis by Fergusson and colleagues of 
aprotinin in cardiac surgery found a dramatically beneficial result 
in the first trial of 22 patients in 1987 (OR for perioperative blood 
transfusion: 0.03, 95% CI: 0.00 to 0.56), with the point estimate 
declining over subsequent randomised trials. By the twelfth study 
(published in June 1 992), the cumulative effect estimate for the 
odds ratio had stabilized (OR: 0.25, P<0.000001) with a total of 
approximately 2500 patients randomised. Throughout the cumu- 
lative meta-analysis, the upper limit of the confidence interval did 
not go higher than 0.65, and in this study, the final meta-analyses 
was 0.34 (95% CI: 0.24-0.41) after the publication of a trial in 
June 2002, by which time a total of more than 8000 patients had 
been randomised [13]. 

New trials too small to resolve remaining uncertainties 

Although systematic reviews of existing evidence sometimes 
reveal data that are sufficient to answer research questions, 
cumulative meta-analyses, like that of tranexamic acid [39], have 
also exposed questions that remain unanswered, but continue to 
be addressed in studies that are much too small to resolve the 
remaining uncertainties [15,40-56]. The importance of addressing 
uncertainties was revealed when a systematic review showed that a 
series of 16 under-powered trials conducted over 23 years failed to 
indicate whether steroids given to patients with traumatic brain 
injury reduced or increased their chances of survival. The 
uncertainty was only resolved when a very large trial showed 
reliably that the treatment increased mortality [57]. 

Discussion 

This large collection of cumulative meta-analyses, from across 
health and social care, and covering a wide range of research into 
the effects of interventions, highlights how a review of the existing 
evidence might have helped researchers, practitioners, patients 
and funders make more informed decisions and choices about new 
clinical trials over many decades of research. The importance of 
taking account of earlier research is not a new concept for either 
the design or interpretation of new studies. For example, it was 
highlighted to the British Association for the Advancement of 
Science by Lord Rayleigh in the nineteenth century [58-59] . 
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The cumulative meta-analyses that have been considered 
together for the first time in this review provide cautionary tales 
about how new research might have been designed and 
implemented, when the existing research would have shown it to 
be unnecessary. Other research, taking a different approach to this 
review, has also highlighted this problem. In 20 1 1 , Herbison and 
colleagues reported cumulative meta-analysis of data from 65 
meta-analyses from 18 Cochrane Reviews. They found that it took 
a median of four (interquartile range: 1 to 6) studies to get within 
10% of the final point estimate. However, they noted that 
although their study suggests that, in many cases, only a few trials 
are necessary before getting a reasonably robust answer, it is 
difficult to know which meta-analyses will change further and 
which will not. They conclude "it is still unclear what character- 
istics of early trials will lead to more confidence being placed in the 
results of individual meta-analyses" [35]. 

It is also important to recognise how unreliable initial evidence 
can be [60—61]. An analysis of data from 85,000 meta-analyses 
with binary outcomes in Cochrane Reviews showed that early 
trials tend to overestimate treatment effects [62] . This may result, 
for example, from studies selecting unrepresentative subgroups of 
patients known to have responded favourably to similar interven- 
tions previously, or from excluding patients who have not 
responded [63]; from biased under-reporting of early trials with 
disappointing results [64]; or time-lag bias in the publication of the 
results of trials [65]. As a result, early studies and meta-analyses of 
them may tend to yield inflated estimates of effects and this needs 
to be taken into account in considering proposals for additional 
studies [66]. However, an assessment of the existing evidence is 
still crucial to providing the ethical, scientific and environmental 
justification for proposed new trials. Without such reviews, those 
responsible for proposed new trials cannot make a well-informed 
decision about whether to proceed with them, and they need to 
include careful assessment of the quality and risk of bias in the 
studies being brought together [67]. This should influence the 
interpretation of the results of the cumulative meta-analysis, 
alongside consideration of the possibility that spurious results 
might arise due to cumulative testing. This may require the use of 
statistical techniques such as sequential analysis, and the need to 
consider the statement "Don't Ignore Chance Effects" when 
building from early, positive findings [68] . 

In considering possible limitations of our research, we note that 
systematic reviews of healthcare interventions are subject to the 
impact of selective reporting by researchers, in which whole 
studies might not be published [69] or, even if studies are 
published, particular findings might be excluded because of the 
authors' or editors' opinions about the findings [70]. This 
methodological review could also be subject to such biases, where 
cumulative meta-analyses may have been performed but not 
reported. In contrast to clinical trials [71] or, more recently, 
systematic reviews with health outcomes [72], there is no widely 
available system to register prospectively methodological research, 
such as cumulative meta-analysis. This makes it impossible to 
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determine the extent of selective reporting of cumulative meta- 
analysis, or its potential impact on our conclusions. Furthermore, 
although there were relatively few examples of cumulative meta- 
analyses in which benefits appeared for an intervention after the 
initial trials had shown null or negative results, this is not 
surprising, because awareness of such early results, even without a 
formal meta-analysis, might discourage future research. By 
contrast, while early positive results might lead to the conduct of 
new trials to confirm those results or to test their reproducibility in 
other settings [73]. 

Despite these limitations, however, and given that this review 
found such a breadth of examples, it is likely that our general 
finding is likely to be valid, namely, that there is a substantial 
problem of waste in research resulting from unnecessary duplica- 
tion because existing research has not been reviewed before and 
after new studies are done [4]. 

Conclusions 

This analysis of 50 reports including over 1500 cumulative 
meta-analyses of clinical intervention studies shows that, had 
researchers assessed systematically what was already known, some 
beneficial and harmful effects of treatments could have been 
identified earlier and might have prevented the conduct of the new 
trials. This would have led to the earlier uptake of effective health 
and social care interventions in practice, less exposure of trial 
participants to less effective treatments, and reduced waste 
resulting from unjustified research. 

We do not argue that the conduct of a new trial in the presence 
of apparent certainty from a meta-analysis of existing research is 
necessarily wrong in all circumstances. The new trial might serve 
to fill a gap or resolve a remaining uncertainty for particular types 
of intervention, patient or setting, or seek to assess important 
outcomes that were not measured in the earlier trials [74]. 
However, we do argue that people designing, funding, conducting 
and then interpreting new studies should make their decision to do 
so in light of an up-to-date systematic review and, if possible, meta- 
analysis of existing related research. This would allow them to 
show the ethical, scientific and environmental basis for their 
proposed new trials, and to demonstrate that they would not be 
wasteful and justify their decision [8]. 
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